Skip to content

7.3. Costs

Where can agent cost grow?

  • Repeated model calls inside one loop/task.
  • Large prompts from session history, tools, skills, or runbooks.
  • Expensive judge/evaluation calls.
  • Idle Kubernetes control plane, node, disks, registry, bucket, and observability retention.
  • Retries or failures that repeat non-idempotent/expensive work.

Latency and cost often share the same cause: unnecessary loop steps or context.

Which hard bounds are implemented?

  • A2A requests default to at most 12 model calls (AGENT_A2A_MAX_LLM_CALLS, valid 1-100).
  • The model gateway allows a local burst/refill of 30 requests per minute per instance.
  • MCP and A2A have their own higher local limits.
  • Qwen3 runs on learner-owned compute for the local path.
  • One small single-replica lab replaces a production HA topology.
  • Artifact Registry deletes tagged or untagged image versions older than 30 days while preserving the five most recent versions.

These are safety rails, not a monetary budget. There is no shipped dollar-cost alert or billing export.

How should model cost be calculated?

Use observed input/output tokens from the selected provider/instrumentation and the provider price effective on the run date:

request_cost = input_tokens * input_price_per_token
             + output_tokens * output_price_per_token
             + provider-specific cached/reasoning/tool charges

Do not hard-code prices in the agent or infer token counts from character length. Local Ollama has no API price but consumes machine time, memory, energy, and operator capacity.

How do I attribute cost to a session or tool?

The after-model callback reads usage_metadata from each response and accumulates prompt/completion tokens into session state, which the DatabaseSessionService persists across turns (budget.py). Each turn also emits the running totals as OpenTelemetry span attributes (visible in MLflow traces) and an agentops.tokens counter, tagged by direction, that Prometheus scrapes:

span.set_attribute("agentops.tokens.session.total", input_tokens + output_tokens)
span.set_attribute("agentops.cost.session.estimate", estimate_cost(input_tokens, output_tokens))

The estimate multiplies tokens by AGENT_INPUT_PRICE_PER_1K / AGENT_OUTPUT_PRICE_PER_1K — 0 for local Ollama, your provider's published rate for hosted models. No vendor price is hardcoded. The gateway streaming path reports no usage, so token attribution covers the non-streaming path; that is one reason AGENT_A2A_STREAMING defaults off.

How do I enforce a budget?

Set AGENT_MAX_TOKENS_PER_SESSION. A before-model callback checks the running total and refuses further work once it is spent, returning an actionable message instead of failing silently or running an open-ended bill:

if used >= settings.max_tokens_per_session:
    return LlmResponse(error_code="TOKEN_BUDGET_EXHAUSTED", ...)

The budget works identically on the free local path and the gateway path — it counts tokens, not dollars. Unset, enforcement is disabled.

What does the dashboard measure today?

Request/error rate, latency, and — since token telemetry was added — an agentops_tokens_token_total counter by direction, which supports a token-throughput panel. Traces additionally carry per-session token and cost-estimate attributes. A USD time series still needs a verified price source per model, so a monetary panel stays an explicit opt-in rather than a shipped claim.

Why is the GKE target conditional?

The module uses one zonal Spot e2-standard-2 node, no Cloud NAT, no public load balancer, a 30 GiB boot disk, 1/5 GiB PVCs, Artifact Registry cleanup, and light GCS artifacts. The under-USD-20 target assumes the billing account's GKE free-tier credit covers the zonal management fee and the lab is destroyed when idle.

Without that credit, GKE charges USD 0.10 per cluster-hour for management. Compute, storage, registry, network, GCS, and Vertex remain separate billable SKUs. Spot capacity/price and provider pricing can change; consult current GKE pricing and the OpenTofu plan.

How do you prevent an idle bill?

Prefer local development. Label cloud resources, review the plan, set an external billing budget/alert, record the start time, and teardown immediately after the lab. tofu destroy is destructive and requires explicit review; a non-empty GCS bucket has force_destroy=false to prevent silent data deletion.

What is the cost checkpoint?

For one representative trace, record model calls, input/output tokens if available, total latency, model path, and current price source/date. For the GKE plan, list every billable resource and the free-tier assumption. Mark unknown values as unknown; do not claim a monthly total from machine type alone.